• Founding engineer at New Relic
• Software architect specializing in web applications
• Working in APM for 10 years
• Data Science, Machine Learning, and Visualization

All good?

• Does it quantify something we care about?
• Does it give us a qualitative assessment of the current status?
• Does it help us measure differences across similar things, like applications or servers?
• Does it help us identify trends and anomalies?
• Does it reveal underlying patterns or relationships in the data?

How cheaply can we collect the data?

• Is space or bandwidth a premium?
• Can it be reduced by combining values into a single value, or do we have to store every measurement?
• How much screen real estate does it take up?

The smaller the data the more things you can measure.

The less space it takes the more options you have for screen layout.

Can you reduce your measurement in X, \(x_1 .. x_n\), by combining values using a reduce function, \(f(a,b)\)?

\[ X_{ab} = f(a, b)\] \[ X_{cd} = f(c, d)\] \[ X_{abcd} = f(X_{ab}, X_{cd})\]

Space required grows linearly with length of history; a simple time series

• Min and Max: \(min(X) = min\left(min(x_1, x_2), min(x_3, x_4), ... min(x_{n-1}, x_n)\right)\)
• Sum: \(\sum_{i=1}^n x_i = (x_1 + x_2) + (x_3 + x_4) +...(x_{n-1} + x_n)\)
• Mean: \(\bar{x} = \frac{1}{n}\sum_{i=0}^n x_i\)
• Standard Deviation: \(\sigma = \sqrt{\frac{\sum_{i=0}^n x_i^2 - n\bar x }{n - 1}}\)

Examples:

• Histograms (keep N buckets for each time period)
• Median
• Percentiles

The amount of storage required depends on the length of history but also the granularity of the data.

You need N buckets in each time period.

Examples:

• Scatterplots
• Replays

Requires keeping a complete history of every measurement.

Space is proportional to the number of measures.

If response times had a normal distribution we could show everything using simple statistics and estimated percentiles:

Summary Statistics Don't Work on Non-Gassian Data

Geometric Mean

Instead of collecting the sum of the response times, you collect the sum of the log of the response times:

\(GM = exp\left(\frac1n\sum_{i=1}^n ln(t_i)\right)\)

Geometric Standard Deviation

Instead of collecting the sum of squares to calculate the standard deviations, you collect the sum of the squared log of response times:

\(GSD = exp\left(\sqrt{\frac1n\sum_{i=1}^nln(t_i)^2-\left(\frac1n\sum_{i=1}^nln(t_i)\right)^2}\right)\)

Inner Quartile Interval Estimate

\(\left[\frac{GM}{GSD^Z}\dotsb GM\times GSD^Z\right]\)

…where \(z = 0.674\) from the Standard Normal Distribution.

• An Apdex score is a standardized measure indicating whether your site is meeting it's obligations
• Values from 0 (total failure) to 100 (every request succeeds)
• Scored based on responses bucketed according to response time thresholds:
  • Satisfied
  • Tolerating
  • Failed
• Timeouts and errors automatically placed in Failed bin.

The Apdex Score is determined by the formula:

\[\frac{N_s + (1/2 * N_t)}{N}\]

Where:

• \(N_s\) = The number of requests completed in no more than \(T\) seconds
• \(N_t\) = The number of requests completed between \(T\) and \(4T\)
• \(N_f\) = The number of requests taking longer than \(4T\), including requests that had errors or timed out (ignored by the formula).
• \(N\) = \(N_s + N_t + N_f\) = the total number of requests processed.

\(T\) is referred to as the Apdex T parameter and is set individually for each application based on what is considered a satisfactory response time from a business perspective.

Apdex Scores for a group of apps can be enumerated with guages, but a richer visual is the "Apdex Button".

• Each application's status is represented with its own "Traffic Light"
• Green → Good
• Red → Bad

Looking at collection of related things

• Cluster of servers
• Application server nodes
• Pages in an application
• Response times broken down by browser

Plots of throughput on the X axis and latency on the Y axis. Color is time of day.

Traffic Animation

Bill Kayser (@bravoking) - www.newrelic.com

• This talk: github.com/bkayser/apmviz
• D3/Javascript version live demo: http://marlowe.datanerd.us
• D3/Javascript source: github.com/newrelic/marlowe
• R Library for getting New Relic Data: github.com/bkayser/NewRelicR